Method for detecting vehicle and device for executing the same

ABSTRACT

There is provided a method for detecting a vehicle including receiving continuously captured front images, setting a search area of the vehicle in a target image based on a location of the vehicle or a vehicle area detected from a previous image among the front images, detecting the vehicle in the search area according to a machine learning model, and tracking the vehicle in the target image by using feature points of the vehicle extracted from the previous image according to a vehicle detection result based on the machine learning model. Since the entire image is not used as a vehicle detection area, a processing speed may be increased, and a forward vehicle tracked in an augmented reality navigation may be continuously displayed without interruption, thereby providing a stable service to the user.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the priority and benefit of Korean PatentApplication No. 10-2018-0156746 filed on Dec. 7, 2018, and10-2019-0151276 filed on Nov. 22, 2019, with the Korean IntellectualProperty Office, the disclosure of which is incorporated herein byreference.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to a method for detecting a vehicle and anelectronic device for executing the method.

2. Description of the Related Art

With the Internet open and laws related to location data modified, thelocation-based service (LBS) related industry has been activated. As afield of the LBS, vehicle navigation related industries for locating acurrent location of a vehicle equipped with a device or guiding amovement route to a destination have been activated and vehicle videorecorder industry for recording driving images of the vehicle todetermine a cause of an accident or an event which occurs has also beenactivated, so vehicles are increasingly equipped with a digital videorecorder or a dashboard camera.

Recently, in addition to a primary function of guiding a route andcapturing driving images, application technologies for assisting adriver in driving a vehicle on the basis of various image processingtechnologies have been developed, and specifically, advanced driverassistance system (ADAS) has been developed and applied to analyzeacquired images through various sensors installed in a vehicle anddetect objects included in the images to determine driving relatedinformation and provide the determined information to a driver.

The ADAS provides guide information to the driver according to a varietyof situations, as one the ADAS, a forward vehicle collision warningsystem (FVCWS) providing guide information to allow a driver torecognize a forward vehicle and maintain an appropriately safe distancewith respect to the forward vehicle has been developed and applied.

The FVCWS may recognize in advance a possibility of a collision when ahost vehicle is close to a forward vehicle by a predetermined distancewhile driving, a possibility of a collision with the forward vehicle ifthe host vehicle maintains a current speed, or an occurrence of anaccident due to an accidental factor that may be caused by the forwardvehicle and provide corresponding warning information. In addition,technologies for providing driving guide information to the driver onthe assumption of recognition of a forward vehicle such as a function ofproviding information on the departure of the forward vehicle in a statein which the host vehicle is waiting for the light to change or the hostvehicle is stopped, besides the collision with the forward vehicle hasalso been continuously developed.

Therefore, in order to increase accuracy of these technologies, it isessential to accurately detect the forward vehicle from the captureddriving images.

SUMMARY

An aspect of the present invention may provide a method of stably andaccurately detecting a forward vehicle. An aspect of the presentinvention may also provide an electronic device supporting safe drivingbased on a detected forward vehicle and a method of guiding forwardvehicle collision of the electronic device.

According to an aspect of the present invention, a method for detectinga vehicle may include: receiving continuously captured front images;setting a search area of the vehicle in a target image based on alocation of the vehicle or a vehicle area detected from a previous imageamong the front images; detecting the vehicle in the search areaaccording to a machine learning model; and tracking the vehicle in thetarget image by using feature points of the vehicle extracted from theprevious image according to a vehicle detection result based on themachine learning model.

The search area may be enlarged and set based on the vehicle areadetected from the previous image.

The search area may be enlarged and set according to a size of thedetected vehicle.

The tracking may include tracking the vehicle by extracting the featurepoints of the vehicle from the vehicle area detected from the previousimage.

The location of the vehicle may be tracked from the target image usingthe extracted feature points of the vehicle when the vehicle detectionbased on the machine learning model fails or the reliability of thedetected vehicle is below a reference.

The tracking may include tracking the vehicle in parallel with thevehicle detection in the detecting, and terminating the tracking of thevehicle when the vehicle detection based on the machine learning modelis successful or when the reliability of the detected vehicle is abovethe reference.

The method may further include displaying the detected or trackedvehicle according to a predetermined user interface.

The displaying may include displaying a forward vehicle collisionrelated notification based on the vehicle according to the predetermineduser interface.

The detecting may further include obtaining a motion vector of thefeature points of the vehicle from a plurality of previous images andgenerating a modified search area based on the motion vectors and thesearch area and include detecting the vehicle from the modified searcharea according to the machine learning model.

The motion vector may be generated based on a relationship betweenpositions at which the feature points of the vehicle are expressed ineach of the plurality of previous images.

A center position of the modified search area may be determined based ona center position of the search area and the motion vector, and a widthof the modified search area may be determined based on a direction orsize of the motion vector.

According to another aspect of the present invention, a vehicledetecting apparatus may include: an image input unit receivingcontinuously captured front images; an area setting unit setting asearch area of a vehicle in a target image based on a location of thevehicle or vehicle area detected from a previous image among the frontimages; a vehicle detecting unit detecting the vehicle from the searcharea according to a machine learning model; and a vehicle tracking unittracking the vehicle in the target image using feature points of thevehicle extracted from the previous image according to a vehicledetection result based on the machine learning model.

The search area may be enlarged and set based on the vehicle areadetected from the previous image.

The search area may be enlarged and set according to a size of thedetected vehicle.

The vehicle tracking unit may track the vehicle by extracting featurepoints of the vehicle from the vehicle area detected from the previousimage.

The vehicle tracking unit may track the location of the vehicle from thetarget image using the extracted feature points of the vehicle when thevehicle detection based on the machine learning model fails or areliability of the detected vehicle is below a reference.

The vehicle tracking unit may track the vehicle in parallel with thevehicle detection in the detecting process and terminate the tracking ofthe vehicle when the vehicle detection based on the machine learningmodel is successful or when a reliability of the detected vehicle isabove the reference.

The vehicle detecting apparatus may further include: an output unitdisplaying the detected or tracked vehicle according to a predetermineduser interface.

The output unit may display a forward vehicle collision relatednotification based on the vehicle according to the predetermined userinterface.

According to another aspect of the present invention, a method ofwarning a vehicle rear-end collision may include: receiving continuouslycaptured front images; setting a search area of a vehicle in a targetimage based on a location of the vehicle or a vehicle area detected froma previous image among the front images; detecting the vehicle in thesearch area according to a machine learning model; tracking the vehiclein the target image by using feature points of the vehicle extractedfrom the previous image according to a vehicle detection result based onthe machine learning model; and determining a possibility of rear-endcollision according to a distance to the detected or tracked vehicle anda relative speed.

Technical solutions of the present invention may not be limited to theabove, and other technical solutions of the present invention will beclearly understandable to those skilled in the art to which the presentinvention pertains from the disclosures provided below together withaccompanying drawings.

According to the present invention, since the entire image is not usedas a vehicle detection area, a processing speed may be increased, and aforward vehicle tracked in an augmented reality navigation may becontinuously displayed without interruption, thereby providing a stableservice to the user.

In addition, by applying the forward vehicle collision warning system(FVCWS) of the intelligent advanced driving assistance system (ADAS), aprocessing rate of detecting and tracking a vehicle based on the machinelearning in detecting a forward vehicle using learned vehicleinformation may be improved.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating an electronic device according toan exemplary embodiment of the present invention.

FIG. 2 is a flowchart illustrating a vehicle detecting method accordingto an exemplary embodiment of the present invention.

FIGS. 3 through 4C are flowcharts illustrating a vehicle detectingmethod in detail according to an exemplary embodiment of the presentinvention.

FIGS. 5 through 7 are exemplary diagrams illustrating setting of anadaptive search area according to an exemplary embodiment of the presentinvention.

FIG. 8 is an exemplary diagram illustrating vehicle tracking based onvehicle detection and feature point extraction based on a machinelearning model in a vehicle detecting method according to an exemplaryembodiment of the present invention.

FIG. 9 is a block diagram illustrating a system according to anexemplary embodiment of the present invention.

FIG. 10 is a diagram illustrating a system network connected to a systemaccording to an exemplary embodiment of the present invention.

FIGS. 11A and 11B are diagrams illustrating a forward vehicle collisionwarning guide screen of a system according to an exemplary embodiment ofthe present invention.

FIG. 12 is a diagram illustrating an implementation form when a systemdoes not have an imaging unit according to an exemplary embodiment ofthe present invention.

FIG. 13 is a diagram illustrating an implementation form when a systemincludes an imaging unit according to an exemplary embodiment of thepresent invention.

FIG. 14 is a diagram illustrating an implementation form using a head-updisplay (HUD) according to an exemplary embodiment of the presentinvention.

FIG. 15 is a block diagram illustrating a configuration of an autonomousvehicle according to an exemplary embodiment of the present invention.

FIG. 16 is a block diagram illustrating a detailed configuration of acontrol device according to an exemplary embodiment of the presentinvention.

DETAILED DESCRIPTION

The following description illustrates only a principle of the presentinvention. Accordingly, a person skilled in the art may invent variousapparatuses implementing the principle of the invention and included ina concept and a scope of the invention even though the apparatuses arenot clear described or illustrated in the present specification. Itshould be further understood that in principle, all conditional termsand exemplary embodiments which are described in the specification areapparently intended to understand the concept of the invention but thepresent invention is not limited to the exemplary embodiments and statesspecifically described in the specification.

The aforementioned objects, characteristics, and advantages will be moreapparent through the detailed description below related to theaccompanying drawings, and thus those skilled in the art to which thepresent invention pertains will easily carry out the technical spirit ofthe present invention.

Further, in the following description, a detailed explanation of apublicly known technology related to the invention may be omitted whenit is determined that the detailed explanation may unnecessarily obscurethe subject matter of the present invention, Hereinafter, an exemplaryembodiment according to the present invention will be described indetail with reference to the accompanying drawings.

The above purpose, characteristics, and advantages of the presentinvention will now be described more fully with reference to theaccompanying drawings. The invention may, however, be embodied in manydifferent forms and in what follows, particular exemplary embodiments ofthe invention are illustrated in the accompanying drawings and describedin detail. Basically, the same reference numbers across the disclosurerepresent the same constituting elements. In addition, if it isdetermined that disclosure related to the invention or a specificdescription about structure of the invention may lead tomisunderstanding of the purpose of the invention, the correspondingspecific description would be omitted.

Hereinafter, a configuration of an electronic device and a serverrelated to the present invention will be described below in more detailwith reference to the accompanying drawings. In the followingdescription, suffixes “module” and “unit” are used only in considerationof facilitating description and do not have meanings or functionsdiscriminated from each other.

The electronic device described in this disclosure may include acellular phone, a smartphone, a notebook computer, a digitalbroadcasting terminal, a personal digital assistant (PDA), a portablemultimedia player (PMP), a navigation terminal, and so on. Hereinafter,the electronic device is assumed to be a navigation terminal.

The traffic-related image, which is a traffic image collected from auser device and other devices (e.g., CCTV, etc.), may be image dataincluding and a still image and video including road congestioninformation, road surface state information, accident information, aroad traffic message (RTM), etc.

Hereinafter, exemplary embodiments of the present invention will bedescribed in detail with reference to the accompanying drawings.

An electronic device 10 according to an exemplary embodiment of thepresent invention will be described with reference to FIG. 1.

FIG. 1 is a block diagram of an electronic device 10 according to anexemplary embodiment of the present invention.

In the present exemplary embodiment, the electronic device 10 mayinclude an image input unit 12, an area setting unit 14, a vehicledetecting unit 16, and a vehicle tracking unit 18.

The image input unit 12 receives an image captured while a vehicle isdriving.

The image input unit 12 may receive a front image directly captured by acamera module included in the electronic device 10. Alternatively, animage related to driving of the vehicle may be directly received from atleast one external camera (not shown). For example, when the electronicdevice 10 operates as a vehicle navigation device, the image input unit12 may receive an image captured by a video recorder device of thevehicle.

Specifically, in the present exemplary embodiment, the image acquired bythe image input unit 12 may be a video which is continuously captured.That is, front images may be received at a predetermined ratio of framesper second. In addition, the frame rate for receiving the front imagesmay be changed according to various conditions such as a speed of thevehicle, weather of a surrounding environment, and the like.

The area setting unit 14 sets a search area of the vehicle in a targetimage based on the location of the vehicle or the vehicle area detectedfrom the previous image among the front images.

The area setting unit 14 previously sets an area in which the vehicle isexpected to exist so that the vehicle detecting unit 16 may detect thevehicle in the set area, instead of searching for the vehicle in theentire input front image (to be described later).

Specifically, the area setting unit 14 may use information of thevehicle detected from the previous image.

In general, if images are input at a rate of about 30 frames per secondand if the forward vehicle does not move very quickly although a speedof the vehicle is counted, the forward vehicle may still be included ina predetermined area based on the position present in the previousimage.

Therefore, the area setting unit 14 may set a candidate area fordetecting the vehicle of the next image based on the location of thevehicle detected from the previous image.

The candidate area may also be set using the detected area of thevehicle.

Specifically, in the present exemplary embodiment, a size of the forwardvehicle may be determined through the detected area of the vehicle, andthe search area may be set by extending a width and a height based onthe size of the vehicle.

The vehicle detecting unit 16 detects the vehicle in the set search areaaccording to a machine learning model.

Specifically, the vehicle may be detected by a method of learning a Haaralgorithm as a method of generating an adaptive detection area in orderto increase a processing rate of vehicle detection, and after thedetection, the detected area of the vehicle may be expanded and used asa detection area of a next image.

In the present exemplary embodiment, the Haar algorithm, which is basedon a theory that a specific feature has a difference between light andshade, used as the learning method is a method of finding a featureusing a difference in brightness between areas in an image, and inpresent exemplary embodiment, a characteristic brightness difference ofthe vehicle may be learned, based on which a feature may be located todetect the forward vehicle.

In the present exemplary embodiment, as the algorithm used for learningto detect the forward vehicle, various image processing methods otherthan the Haar algorithm may be used. For example, histogram of orientedgradient (HOG) and local bit pattern (LBP) feature point extractingmethods may be used. In addition, Adaboost and support vector machine(SVM) algorithm may be applied as the learning algorithm.

However, in the detection method according to the present exemplaryembodiment, vehicle detection may be continuously performed ceaselesslythrough feature point-based vehicle tracking in case of non-detection inorder to compensate for a problem that may occur when the vehicletracking is performed only by the detection based on the machinelearning.

Therefore, if the vehicle tracking unit 18 fails to detect the vehiclein the search area set in the target image, the vehicle tracking unit 18may extract a feature point from the previous front image or the vehiclearea of the previous front image and track the vehicle area using anoptical flow.

Here, referring to FIG. 8, in the present exemplary embodiment, thevehicle detecting unit 16 may continue to detect the vehicle based onthe machine learning simultaneously when the vehicle tracking unit 18tracks the vehicle area.

That is, the tracking process of the vehicle tracking unit 18 is stoppedwhen the vehicle detecting unit successfully detects the vehicle, anadaptive search area is set with the detected vehicle area, and vehicledetection is continued.

In addition, the vehicle tracking unit 18 may track movement of theforward vehicle using the optical flow.

Hereinafter, a vehicle detecting method according to an exemplaryembodiment of the present invention will be described in more detailwith reference to FIGS. 2 through 4.

FIG. 2 is a flowchart illustrating a vehicle detecting method accordingto the present exemplary embodiment.

Referring to FIG. 2, the image input unit receives continuously capturedfront images (S10).

The image input unit 12 receives the front images captured through thecamera module according to a determined frame rate.

For example, the front images captured at a rate of 30 frames per secondmay be continuously received, and if a frame rate for receiving theimages is set, the captured front images may be received according tothe set frame rate.

The area setting unit 14 sets a search area of the vehicle in a targetimage based on a location of the vehicle or a vehicle area detected in aprevious image among the continuously input front images (S20).

In the present exemplary embodiment, the previous image may be at leastone image of the input front images and may include a plurality ofimages as well as a single image.

In addition, the target image may be an image which is a target ofextraction of an object among the input front images.

In the present exemplary embodiment, when the vehicle is detected in theprevious image, the area setting unit 14 may set the search area basedon the location of the vehicle or the area recognized as the vehicle,and apply the set search area to the target image.

Specifically, referring to FIG. 3, when the front image is input, thearea setting unit 14 may first determine whether an adaptive search areaexists (S22).

When the vehicle detection in the previous front image is successful,the area setting unit 14 may adaptively expand the search area accordingto information of the vehicle detected in the previous front image togenerate a search area and set the generated search area to an area inwhich the vehicle is to be searched from a next target front image.

If the adaptive search area exists, the search area may be set to anarea for detecting the vehicle (S24).

When the search area is set, the vehicle detecting unit 16 detects thevehicle from the search area according to the machine learning model(S30).

Referring to FIG. 3, in the detecting of the vehicle (S30), the vehicleis detected from the search area of the front image. Image informationof the forward vehicle may be searched according to the machine learningmodel, an object corresponding thereto may be detected, and the objectmay be recognized as the forward vehicle.

Further, in the present exemplary embodiment, when setting a candidatearea for detecting the vehicle, the candidate area may be set using adetected lane (e.g., lane marking) marking (S23). That is, in thepresent exemplary embodiment, when setting the candidate area fordetecting the vehicle in the front image, the vehicle may be efficientlyand rapidly detected by setting a driving lane (e.g., lane marking)based on a lane (e.g., lane marking) marking detected as a range of thearea as a detection target area.

Since the lane (e.g., lane marking) marking may be defined in a simple,uniform shape compared to the vehicle, the lane (e.g., lane marking) maybe more easily detected from the image and the vehicle detecting unit 16may also detect the vehicle using the detected lane (e.g., lane marking)marking (S30).

Since the vehicle detected in the present exemplary embodiment may be aforward vehicle as a target, the lane (e.g., lane marking) marking maybe used such that the vehicle in the traffic lane (e.g., lane marking)in which the vehicle is currently driving may be detected as the forwardvehicle.

In general, since the vehicle exists in the detected traffic lane (e.g.,lane marking), the area setting unit 14 may determine the setting of thesearch area into the driving lane (e.g., lane marking).

Furthermore, in the present exemplary embodiment, the adaptive searcharea may be adaptively enlarged and set in consideration of a size ofthe vehicle, and thus may be used to consider a ratio of the vehicle tothe lane (e.g., lane marking) to determine the size of the vehicle.

Since a width of the lane (e.g., lane marking) may be standardized anddetermined according to a standard of a general road in advance, a typeof the vehicle may be determined based on the ratio of the width of thevehicle to the determined width of the lane (e.g., lane marking) and maybe used for setting the search area.

When there is no adaptive search area generated in advance, the vehicledetecting unit 16 may detect the vehicle according to the machinelearning model from the front image without setting the search area(S30).

For example, when the user activates a function of detecting the forwardvehicle according to the present exemplary embodiment, since a previoussearch area has not been generated at a first activation time, theforward vehicle may be detected using the entire front image.

Alternatively, as described above, the search area may be partiallylimited by using the information of the lane (e.g., lane marking)marking in the front image and may be used to first detect the forwardvehicle.

Alternatively, a road area may be recognized based on a vanishing pointof the front image, and the forward vehicle may be detected in the area.

In addition, in a case where a search area is not present and a vehicleis detected from a first forward image without a search area, abackground area and a road area may be distinguished from the entireimage, and for the road area, a traffic lane (e.g., lane marking) inwhich the vehicle drives may be first detected and then an objectlocated in a traffic lane (e.g., lane marking) may be detected as aforward vehicle.

When the vehicle detecting unit 16 detects a vehicle from the entirefirst front image, the area setting unit 14 may set a search area of atarget image in which a vehicle is to be detected using the adaptivesearch area generated based on the vehicle detected from the previousimage.

Further, in present exemplary embodiment, the vehicle detecting unit 16may determine whether the vehicle is detected.

That is, even if the search area exists, a case may occur in which thevehicle is not detected due to other optical factors, changes in asurrounding environment, or other errors.

Therefore, according to the vehicle detecting method of the presentexemplary embodiment, vehicle detection may continue through vehicletracking if a forward vehicle is not detected.

The detection of the forward vehicle may be performed as an auxiliaryfunction for safe driving, such as a front collision situation. Ifdetection of a forward vehicle, which actually exists, fails so it iserroneously recognized that the forward vehicle is not present,erroneous operation information may be provided to the user. Therefore,the present exemplary embodiment proposes a method of additionallyperforming vehicle detection if a vehicle is not detected according tothe machine learning model.

Specifically, referring to FIG. 4A, in the detecting of a vehicle (S30),it is determined whether a forward vehicle is successfully detected(S32), and if the detection is successful, a search area may beadaptively generated through information of the detected vehicle (S34).

However, if the detection of the vehicle based on the machine learningmodel fails, the vehicle tracking unit 18 may continue to track thevehicle using vehicle feature points.

Therefore, in the tracking (S40), if the vehicle detection based on themachine learning model fails, the vehicle tracking unit 18 may firstdetermine whether there is a feature point from the previously extractedimage of the vehicle (S42).

In the present exemplary embodiment, the feature point may be an areaincluding one pixel or a plurality of pixels of an opticallycharacterized image. For example, an area having a brightness greaterthan a reference value or having a feature value that may bedistinguished in comparison with surrounding objects in the image of theforward vehicle detected up to a recent time.

In addition, the feature point may be set to a value which is changedless in the vehicle area of the continuously input front images.

That is, the vehicle tracking unit 18 may set an area which isdistinguished from other peripheral objects and which has a featurevalue changed less in the vehicle area.

If a feature point that satisfies the above condition is set in advance,the vehicle area may be tracked using an optical flow of the featurepoint (S44).

If a predetermined feature point does not exist, a feature point may beextracted in an adaptive search area of the previous front image (S46)and the feature point extracted from the previous image may be trackedaccording to an optical flow in a subsequently input front image torecognize a vehicle (S44).

The process of detecting or tracking the vehicle described above may berepeatedly performed according to a continuous input of the front image(S10).

In present exemplary embodiment, the vehicle tracking unit 18 maydetermine when vehicle detection fails by comparing the detectedreliability of the vehicle with a reference value. That is, even if thevehicle is detected by the machine learning model, when the reliabilityis low compared to the previous image of the vehicle, the vehicle may betracked using the feature point.

In the present exemplary embodiment, the reliability is used as areference for determining whether the vehicle tracking unit 18 performsvehicle tracking, and the vehicle tracking unit 18 may determine whetherto perform vehicle tracking by determining whether an object detected bythe vehicle detecting unit 16 from the image is a vehicle.

For example, the object in the image detected by the vehicle detectingunit 16 may be compared with a predetermined vehicle criterion (e.g.,size, shape, color or a conformity degree to an object in the previousimage, etc.) to determine that the object is a forward vehicle, and adegree to which the object matches the criteria may be determined asreliability of the recognized object. Therefore, when the reliability ofthe detected object is less than 50%, for example, the vehicle trackingunit 18 may determine that the vehicle detection fails, and performvehicle tracking.

In addition, in the present exemplary embodiment, the vehicle detectingunit 16 may directly determine the reliability of the detected object inthe image. Therefore, the vehicle detecting unit 16 may determine thereliability of the detected object with respect to the vehicle, and ifthe reliability is lower than the reference, the vehicle detecting unit16 may deliver a result of determination that the vehicle detectionfails to the vehicle tracking unit 18 so that vehicle tracking may beperformed by the vehicle tracking unit 18.

However, in the vehicle detecting method according to the presentexemplary embodiment, even when detection fails in the vehicle detecting(S30) and tracking of the vehicle is performed using feature points(S40), vehicle detection based on the machine learning model (S30) iscontinuously performed.

In other words, in the present exemplary embodiment, performing ofvehicle detection based on the machine learning model is basically set,and vehicle tracking based on an optical flow may be additionallyperformed to recognize movement of the forward vehicle continuously evenwhen vehicle detection based on the machine learning model fails.

Therefore, if the vehicle detection fails while continuously performingthe vehicle detection based on the machine learning model, vehicletracking based on the optical flow may be additionally performed, and ifvehicle detection again based on the machine learning model issuccessful, the vehicle tracking based on the optical flow may bestopped and only the vehicle detection operation based on the machinelearning model may be performed.

In addition, in the present exemplary embodiment, when the detecting ofthe vehicle (S30) is performed together with the tracking of the vehicle(S40), a tracking result of the vehicle may be further used.

That is, the detecting of the vehicle (S30) may use the information ofthe feature point determined in the process of tracking the vehicle whenthe adaptive search area is set to increase a success rate of thevehicle detection.

Specifically, in the detecting of the vehicle (S30), a motion vector ofthe feature point of the vehicle may be obtained from a plurality ofprevious images.

That is, the area setting unit 14 may calculate an optical flow as amotion vector and reflect the motion vector in the adaptive search areagenerated in the previous image.

Therefore, in the setting of the area (S20), a modified search area maybe generated based on the motion vector and the search area.Specifically, an expansion value of the search area may be determinedaccording to a size and direction of the motion vector and a modifiedsearch area may be generated.

Thereafter, in the detecting of the vehicle (S30), vehicle detection maybe performed in the modified search area based on the machine learningmodel and a detection probability of the vehicle may be furtherincreased.

In addition, when the vehicle is detected based on the machine learningmodel in the detecting the vehicle (S30) in the present exemplaryembodiment, a classifier separately trained according to a size of thevehicle or a characteristic of a distance to the forward vehicle mayalso be used.

Specifically, in order to increase vehicle detection performance basedon the machine learning, classifiers generated after classifyingdatabases storing images of the vehicle into a general passenger vehicleand a large vehicle and learning the same may be used.

In addition, in order to increase short-range vehicle detection, aclassifier which has trained only an image of a tail light side, ratherthan the entire vehicle image, may also be used.

Referring to FIG. 4B, in present exemplary embodiment, three classifiersmay be trained.

For example, a first classifier 16 a may be trained using a firstvehicle image 12 a.

Specifically, the first vehicle image 12 a is an image of a generalpassenger vehicle, and the first classifier 16 a may perform the machinelearning using the image of the general passenger vehicle and improveclassification performance for the general passenger vehicle.

A second classifier 16 b may be trained using a second vehicle image 12b.

In this case, unlike the first vehicle image 12 a, the second vehicleimage 12 b may be a rear image of a vehicle such as a bus or a truck asa large vehicle.

Therefore, the second classifier 16 b trained with the second vehicleimage 12 b may have high classification performance for a large vehicle.

In addition, a third classifier 16 c may be separately trained inconsideration of a feature based on a distance to a forward vehicle, notthe size of the vehicle.

That is, a third vehicle image 12 c obtained for a vehicle located verynearby may include only information on a partial area, rather than theentire image of the vehicle such as the first vehicle image 12 a and thesecond vehicle image 12 b.

Therefore, in the present exemplary embodiment, the third classifier 16c may be trained using the separate image including only the partialarea as a learning image.

Through the above process, the vehicle may be detected more accuratelyusing the classifiers individually trained according to the imagefeatures based on the size of each vehicle or the distance to theforward vehicle.

A detailed method of detecting a vehicle will be described in moredetail with reference to FIG. 4C.

When detection is performed using a machine-trained classifier in thedetecting of the vehicle (S30) described above, a plurality ofindividually classified classifiers may be sequentially used.

Therefore, first, detection based on the first classifier is performed(S1000).

A detection result is checked (S1010) and if a first vehicle isclassified by the first classifier trained according to the firstvehicle image, the detecting of the vehicle (S30) may be terminatedbecause the detection is successful (S5000).

However, if the first vehicle is not detected, detection based on thesecond classifier may be performed (S2000).

A detection result is checked (S2010) and if a second vehicle isdetected by the second classifier trained according to the secondvehicle image, the detecting of the vehicle (S30) may be terminatedbecause the detection is successful (S5000). However, if the secondvehicle is not detected even by the second classifier, detection may beperformed by the third classifier (S3000).

Finally, when a vehicle is detected by the third classifier (S3010), thedetecting of the third vehicle located nearby (S3010) may be terminatedbecause the detection is successful (S5000).

If a vehicle is not detected even by the third classifier, since thedetection fails (S4000), a process of tracking a vehicle based on afeature point may be performed in the tracking of the vehicle (S40).

In addition, in the above exemplary embodiment, it is described that thedetection process according to each classifier is sequentiallyperformed, but when parallel processing is possible, inputting to theclassifier may be performed at the same time and the classificationresults may be collected to generate an optimal detection result.

In the case of a vehicle close nearby such as the third classifier, itis necessary to preferentially detect the vehicle according to safetyrequirements, so that the detection based on the third classifier may beperformed with priority.

Hereinafter, setting of an adaptive search area according to the presentexemplary embodiment will be described in more detail with reference toFIGS. 5 through 8.

FIG. 5 is a diagram illustrating an example of detecting a vehicle froman input front image 1000 according to an exemplary embodiment.

Referring to FIG. 5, the vehicle detecting unit 16 may detect a vehicle1200 from an input front image 1000. In present exemplary embodiment,the vehicle is detected from an adaptive search area, but as describedabove, if there is no search area generated as the first front image,the vehicle may be detected from the entire front image.

For example, in the case of a first time point at which a vehicledetection function is activated, the adaptive search area is not set, sothat the object determined as the forward vehicle may be detected in theentire image.

In addition, the vehicle detecting unit 16 may more easily detect theforward vehicle using lane (e.g., lane marking) information 1105 in aroad as described above.

When the vehicle is detected in the above process, the vehicle detectingunit 16 may calculate width w and height h information of an object 1200determined as the vehicle. The area setting unit 14 may set the adaptivesearch area using the calculated width and height.

Referring to FIG. 6, in the present exemplary embodiment, the areasetting unit may expand a search area according to height and widthvalues of the vehicle in an area 1100 in which the vehicle 1200 isdetermined to exist in a previous image to adaptively generate a searcharea 1300 in the front image.

In the present exemplary embodiment, the area expanded in fourdirections by half (w/2) of the width and half (h/2) of the height isset as the adaptive search area.

The area setting unit 14 may set the adaptive search area on theassumption that the vehicle will be located in the expanded adaptivesearch area unless there is a sudden change in the speed of the vehiclein the continuously input image in consideration of a frame rate.

For example, if an image received in the image input unit 12 is 30frames per second, it is included in an area range of a previous imageunless movement of the forward vehicle is very large.

Therefore, when the vehicle is detected in the first image in which theforward vehicle first appears, a search area enlarged according to thedetected vehicle width or height is set, and this area is designated asa search area for vehicle detection in a next successive image. If thisprocess is repeatedly performed, the search area is adaptively changedaccording to the size or location of the detected vehicle.

The adaptive search area set in the present exemplary embodiment isbased on the size of the vehicle, but the size may be determined using aframe rate of the input image, and in addition, the information such asthe speed of the vehicle, a driving direction of the vehicle, and thelike may also be additionally considered.

For example, when a speed difference with the forward vehicle is large,the adaptive search area may be further enlarged, and the search areamay be enlarged even when the vehicle is making a curve.

Furthermore, in the present exemplary embodiment, if the curve is asudden curve with a predetermined curvature or greater, a vehicle in anext lane (e.g., lane marking) may be recognized as a forward vehicle.Therefore, it is also possible to accurately recognize the vehiclelocated in the current driving lane (e.g., lane marking) as a forwardvehicle by utilizing a lane (e.g., lane marking) detection resulttogether with the adaptive search area.

As described above, when the adaptive search area 1300 is set accordingto the area 1100 in which the vehicle 1200 exists as illustrated in FIG.7, the vehicle detecting unit 16 may detect the vehicle in a partialarea of the input front image 1000.

When the search area of the front image used for detecting a firstforward vehicle in FIG. 5 is compared with the search area of the frontimage in FIG. 7, the candidate area 1300 in which a forward vehicle isexpected to exist is set and vehicle detection is performed only in theset range according to the machine learning model in FIG. 7, the forwardvehicle may be more rapidly and accurately detected.

Meanwhile, if the vehicle detection fails in the search area set in thefront image as described above, a feature point is extracted from aprevious vehicle area and a vehicle area is tracked using an opticalflow, and at the same time, vehicle detection based on the machinelearning is also performed.

When a vehicle is detected again in the front image using the machinelearning model, vehicle tracking using the feature point is stopped andvehicle detection continues to be performed in the search area.

Referring to FIG. 8, a vehicle is detected by continuously setting anadaptive search area according to the machine learning model, and when avehicle is not detected from a front image, a process of tracking avehicle through feature point extraction may be simultaneously performed(82).

The vehicle detecting method according to the present exemplaryembodiment described above has the advantage that a processing speed isfaster because the entire image is not used as a vehicle detection areaand the forward vehicle tracking in an augmented reality navigation maybe continuously displayed to thereby provide a stabilized service to theuser.

Meanwhile, such an electronic device 10 may be implemented as one moduleof an advanced driver assistance system (ADAS) or a system 100 forautonomous driving to perform route guide and forward vehicle collisionwarning system (FVCWS). This will be described in more detail withreference to FIGS. 9 and 10.

FIG. 9 is a block diagram illustrating a system according to anexemplary embodiment of the present invention. Referring to FIG. 9, thesystem 100 includes all or some of a storage unit 110, an input unit120, an output unit 130, a curve guide unit 140, an augmented reality(AR) providing unit 160, a controller 170, a communication unit 180, asensing unit 190, and a power supply unit 195.

Here, the system 100 may be implemented by various devices such as asmartphone, a tablet computer, a laptop computer, a personal digitalassistant (PDA), a portable multimedia player (PMP), smart glasses,project glasses, navigation which may provide driving-related guide to adriver of a vehicle, a digital video recorder, a car dash cam, or a carvideo recorder which are imaging devices for vehicle, and the like, andmay be provided in a vehicle.

Driving-related guide may include various guide for assisting a driverin driving a vehicle such as route guide, lane (e.g., lane marking)departure guide, lane (e.g., lane marking) keeping guide, forwardvehicle departure guide, traffic light change guide, forward vehiclecollision warning guide, traffic lane (e.g., lane marking) change guide,traffic lane (e.g., lane marking) guide, curve guide, and the like.

Here, the route guide may include augmented reality (AR) route guide toperform route guide by combining various information such as a position,a direction, and the like of the user to a captured image of a frontside of a vehicle in operation and 2-dimensional (2D) or 3-dimensional(3D) route guide to perform route guide by combining various informationsuch as the user's location, direction, and the like to 2D or 3D mapdata.

In addition, the route guide may include an aerial map route guide toperform route guide by combining various information such as a user'slocation, direction, and the like to aerial map data. Here, the routeguide may be interpreted as a concept including not only the route guideof a case where the user gets in a vehicle to drive the vehicle but alsoa route guide of a case where the user walks or runs to move.

In addition, the lane (e.g., lane marking) departure guide mayguide/notify a driving vehicle of whether or not the driving vehicle isout of a lane (e.g., lane marking).

In addition, the lane (e.g., lane marking) keeping guide may guide thevehicle to return to the lane (e.g., lane marking) in which the vehicleis originally driving.

In addition, the forward vehicle departure guide may guide departure ofa vehicle located in front of a vehicle being stopped.

In addition, the traffic light change guide may guide/notify the vehicleof whether or not a signal of a traffic light located in front of thevehicle being stopped is changed. For example, the traffic light changeguide may guide a change in a traffic light from a red traffic lightindicating a stop signal to a blue traffic light indicating a startsignal.

In addition, the forward vehicle collision warning guide may guide thata distance to a vehicle located in front of a vehicle being stopped ordriving is within a certain distance to prevent a collision with theforward vehicle.

Specifically, in the present exemplary embodiment, the distance betweenthe forward vehicle and the current vehicle is calculated through themachine learning model or feature point extraction and a collisionprevention guide may be provided accordingly.

In addition, the traffic lane (e.g., lane marking) change guide mayguide a vehicle from a traffic lane (e.g., lane marking) in which thevehicle is located to another traffic lane (e.g., lane marking) to guidea route to a destination. In the present exemplary embodiment, it isalso possible to additionally determine whether a forward vehicle existsin a traffic lane (e.g., lane marking) to which the current vehicle isto move, and provide corresponding change guide.

In addition, the traffic lane (e.g., lane marking) guide may guide thevehicle in a traffic lane (e.g., lane marking) in which the currentvehicle is located.

In addition, the curve guide may guide/notify the vehicle that a road onwhich the vehicle will drive after a predetermined time is a curve.

A driving-related image, such as a front image of a vehicle that enablesprovision of various guide, may be captured by a camera mounted in avehicle or a camera of a smartphone. Here, the camera may be a cameraformed integrally with the system 100 mounted in the vehicle and imagingthe front of the vehicle.

As another example, the camera may be a camera mounted in the vehicleseparately from the system 100 to image the front of the vehicle. Inthis case, the camera may be a separate vehicle image capturing devicemounted toward the front of the vehicle, and the system 100 may receivea captured image through wired/wireless communication with the vehicleimage capturing device mounted separately or receive a captured image ofthe vehicle image capturing device when a storage medium storing thecaptured image is inserted into the system 100.

Hereinafter, the system 100 according to an exemplary embodiment of thepresent invention will be described in more detail based on the abovedescription.

The storage unit 110 functions to store various data and applicationsrequired for the operation of the system 100. In particular, the storageunit 110 may store data necessary for the operation of the system 100,for example, an OS, a route search application, map data, and the like.In addition, the storage unit 110 may store data generated by theoperation of the system 100, for example, searched route data, areceived image, and the like.

The storage unit 110 may be implemented as an internal storage such as arandom access memory (RAM), a flash memory, a read only memory (ROM),erasable programmable ROM (EPROM), electronically erasable andprogrammable ROM (EEPROM), a register, a hard disk, a removable disk, amemory card, a universal subscriber identity module (USIM), or the like,as well as a removable storage such as a USB memory.

The input unit 120 functions to convert a physical input from theoutside of the system 100 into a specific electrical signal. Here, theinput unit 120 may include all or some of a user input unit 121 and amicrophone unit 123.

The user input unit 121 may receive a user input such as a touch or apush operation. Here, the user input unit 121 may be implemented usingat least one of various types of buttons, a touch sensor receiving atouch input, and a proximity sensor receiving an approaching motion.

The microphone unit 123 may receive a user's voice and sound generatedin and outside the vehicle.

An output unit 130 is a unit for outputting data of the system 100 tothe user, as image and/or sound. Here, the output unit 130 may includeall or some of a display unit 131 and an audio output unit 133.

The display unit 131 is a unit for outputting data which may be visuallyrecognized by the user. The display unit 131 may be implemented as adisplay unit provided on a front surface of a housing of the system 100.In addition, the display unit 131 may be formed integrally with thesystem 100 to output visual recognition data or may be installedseparately from the system 100, such as a head up display (HUD), tooutput visual recognition data.

The audio output unit 133 is a unit for outputting data that may berecognized acoustically by the system 100. The audio output unit 133 maybe implemented as a speaker that expresses data of the system 100 to bereported to the user, as a sound.

A curve guide unit 140 may perform a function of the curve guidance,described above. Specifically, the curve guide unit 140 may obtain linkinformation corresponding to a road on which the vehicle drives,determine a location of the vehicle at a link at a future time point,and determine a risk of a curve section in which the vehicle is to driveafter a predetermined time using the determined location and a vehiclespeed at a reference time point.

The AR providing unit 160 may provide an AR view mode. Here, AR refersto a method of providing additional information (e.g., a graphic elementindicating a point of interest (POI), a graphic element guiding a curve,and various additional information for assisting a driver in drivingsafe, etc.) visually in an overlapping manner on a screen containing areal world that the user actually views.

The AR providing unit 160 may include all or some of a calibration unit,a 3D space generating unit, an object generating unit, and a mappingunit.

The calibration unit may perform a calibration for estimating a cameraparameter corresponding to the camera from an image captured by thecamera. Here, the camera parameter is a parameter constituting a cameramatrix which is information indicating a relationship in which an actualimage space is formed on a picture, and may include extrinsic parametersand intrinsic parameters.

The 3D space generating unit may generate a virtual 3D space based onthe image captured by the camera. Specifically, the 3D space generatingunit may generate a virtual 3D space by applying a camera parameterestimated by the calibration unit to a 2D captured image.

The object generating unit may generate an object for guiding in AR, forexample, a route guide object, a front collision warning guide object, atraffic lane (e.g., lane marking) change guide object, a lane (e.g.,lane marking) departure guide object, a curve guide object, and thelike.

The mapping unit may map an object generated by the object generatingunit to a virtual 3D space generated by the 3D space generating unit.Specifically, the mapping unit may determine a position in the virtual3D space of the object generated by the object generating unit and mapthe object to the determined position.

Meanwhile, the communication unit 180 may be provided for the system 100to communicate with other devices. The communication unit 180 mayinclude all or some of a location data unit 181, a wireless internetunit 183, a broadcast transceiver unit 185, a mobile communication unit186, a short-range communication unit 187, and a wired communicationunit 189.

The location data unit 181 is a device for obtaining location datathrough a global navigation satellite system (GNSS). GNSS refers to anavigation system that may calculate a location of a receiver terminalusing radio signals received from satellites. Specific examples of GNSSinclude global positioning system (GPS), Galileo, global orbitingnavigational satellite system (GLONASS), COMPASS, Indian regionalnavigational satellite system (IRNSS), quasi-zenith satellite system(QZSS), and the like depending on an operating subject. The locationdata unit 181 of the system 100 according to an exemplary embodiment ofthe present disclosure may obtain location data by receiving a GNSSsignal provided in an area where the system 100 is used. Alternatively,the location data unit 181 may obtain location data throughcommunication with a base station or an access point (AP) in addition tothe GNSS.

The wireless Internet unit 183 is a unit for accessing the wirelessInternet to obtain or transmit data. The wireless Internet unit 183 mayaccess the Internet through various communication protocols defined toperform wireless data transmission and reception of wireless LAN (WLAN),wireless broadband (Wibro), world interoperability for microwave access(Wimax), and high speed downlink packet access (HSDPA).

The broadcast transceiver unit 185 is a unit for transmitting andreceiving broadcast signals through various broadcast systems. Broadcastsystems that may transmit and receive signals through the broadcasttransceiver unit 185 include digital multimedia broadcasting terrestrial(DMBT), digital multimedia broadcasting satellite (DMBS), media forwardlink only (MediaFLO), digital video broadcast handheld (DVBH),integrated services digital broadcast terrestrial (ISDBT), and the like.The broadcast signals transmitted and received through the broadcasttransceiver unit 185 may include traffic data, living data, and thelike.

The mobile communication unit 186 may perform voice and datacommunication by accessing a mobile communication network according tovarious mobile communication standards such as 3rd generation (3G), 3rdgeneration partnership project (3GPP), long term evolution (LTE), andthe like.

The short-range communication unit 187 is a unit for near fieldcommunication. As described above, the short-range communication unit187 may perform communication through Bluetooth, radio frequencyidentification (RFID), infrared data association (IrDA), ultra wideband(UWB), ZigBee, near field communication (NFC), and wireless-fidelity(Wi-Fi).

The wired communication unit 189 is an interface unit that may connectthe system 100 to another device by wire. The wired communication unit189 may be a USB module capable of communicating through a USB port.

The communication unit 180 may communicate with another device using atleast one of the location data unit 181, the wireless Internet unit 183,the broadcast transceiver unit 185, the mobile communication unit 186,the short-range communication unit 187, and the wired communication unit189.

As an example, when the system 100 does not have a camera function, animage captured by an image capture unit for a vehicle such as a digitalvideo recorder, a car dash cam, or a car video recorder may be receivedusing at least one of the short-range communication unit 187 and thewired communication unit 189.

As another example, in the case of communicating with a plurality ofdevices, any one may communicate with the short-range communication unit187, and the other may communicate with the wired communication unit119.

The sensing unit 190 is a unit that may detect a current state of thesystem 100. The sensing unit 190 may include all or some of a motionsensing unit 191 and a light sensing unit 193.

The motion sensing unit 191 may detect a motion in a three-dimensionalspace of the system 100. The motion sensing unit 191 may include a3-axis geomagnetic sensor and a 3-axis acceleration sensor. By combiningmotion data obtained through the motion sensing unit 191 with locationdata obtained through the location data unit 181, a trace of a vehicleto which the system 100 is attached may be more accurately calculated.

The light sensing unit 193 is a unit for measuring ambient illuminanceof the system 100. Using the illuminance data acquired through the lightsensing unit 193, brightness of the display unit 131 may be changed tocorrespond to ambient brightness.

The power supply unit 195 is a device for supplying power necessary foran operation of the system 100 or an operation of other devicesconnected to the system 100. The power supply unit 195 may be a batterybuilt in the system 100 or a unit that receives power from an externalpower source such as the vehicle. In addition, the power supply unit 195may be implemented as a wired communication unit 119 or a device that iswirelessly supplied according to a form of receiving power.

The controller 170 controls an overall operation of the system 100.Specifically, the controller 170 may control all or some of the storageunit 110, the input unit 120, the output unit 130, the curve guide unit140, the AR providing unit 160, the communication unit 180, the sensingunit 190, and the power supply unit 195.

In particular, the controller 170 may acquire link informationcorresponding to a road on which the vehicle will drive. Here, the linkinformation may be obtained from route guide data for route guide to adestination.

For example, when destination information is input through the inputunit 120, the controller 170 may generate route guide data to adestination using map data previously stored in the storage unit 110.Alternatively, when destination information is input through the inputunit 120, the controller 170 may transmit a route guide requestincluding at least one of current location information and destinationinformation to the server. The controller 170 may receive route guidedata from the server according to the route guide request. In this case,the controller 170 may obtain link information corresponding to a roadon which the vehicle drives from the route guide data.

In addition, when an estimated driving route information of the vehicleis generated based on real-time location information of the vehicle, thecontroller 170 may obtain link information based on the generatedinformation.

Meanwhile, the controller 170 may provide forward vehicle collisionwarning guide information according to an exemplary embodiment of thepresent invention. That is, the controller 170 may detect a forwardvehicle from an input front image and provide guide informationaccording to a distance of the detected forward vehicle and a speed ofthe current vehicle and the forward vehicle. In this case, thecontroller 170 may additionally calculate the distance and the relativespeed in the determination process of FIGS. 1 to 4.

That is, the controller 170 may calculate the relative speed of theforward vehicle in consideration of a change of the distance to thevehicle in the input front image and a frame rate and generate frontcollision warning guide information by comparing the relative speed ofthe forward vehicle with the speed of the current vehicle.

The controller 170 may control the output unit 130 to output necessarydeceleration information according to a determination result. Inaddition, the controller 170 may calculate acceleration informationnecessary for specific deceleration.

In the present exemplary embodiment, required acceleration A_(req)(t)for deceleration at a current time t may be calculated usingacceleration A_(TV) of the forward vehicle V_(TV), the relative speedV_(r)(t), V_(r)(t)=V_(TV)(t)−V_(SV)(t) between the forward vehicleV_(TV) and the driving vehicle V_(SV), and the distance X_(c)(t) betweenthe forward vehicle and the driving vehicle (See Equation below).

$\begin{matrix}{{A_{req}(t)} = {A_{TV} + \frac{\left( {V_{r}(t)} \right)^{2}}{2^{*}\left( {{x_{c}(t)} - {x_{r}(t)}} \right)}}} & \lbrack{Equation}\rbrack\end{matrix}$

Further, the distance to the forward vehicle may be calculated byfurther considering a driving distance X_(r)(t) for a time required forthe driver to react to control a brake for deceleration.

In addition, in the present exemplary embodiment, in the case of acut-in vehicle which changes lane (e.g., lane marking) to the drivinglane (e.g., lane marking) in addition to the forward vehicle in themovement traffic lane (e.g., lane marking) of the driving vehicle, thecontroller may determine whether to calculate a required accelerationfor forward vehicle collision warning in consideration of a lateraldistance of the driving vehicle and the forward vehicle.

Specifically, when detecting the cut-in vehicle that changes lane (e.g.,lane marking), it may be determined whether to calculate the requiredacceleration by comparing a vehicle width of the driving vehicle with alateral distance between the center lines of both vehicles.

The controller 170 may control the output unit 130 by stages accordingto the required acceleration calculated through the above process.

If the required acceleration is a first level, the controller 170 maycontrol the output unit 130 to output a first deceleration guide. Here,the first level may be a numerical value indicating that the user needsto decelerate preliminarily.

Here, the numerical value representing the need for deceleration may becalculated in consideration of the distance between the driving vehicleand the forward vehicle, the speed of the vehicle, and the number oflane (e.g., lane marking.

If the required acceleration is a second level, the controller 170 maycontrol the output unit 130 to output a second deceleration guide. Here,the second level may be a numerical value indicating that the user needsa higher degree of deceleration.

If the required acceleration is lower than the first level or if thespeed of the driving vehicle is lower than a reference speed, thecontroller 170 may control the output unit 130 not to output adeceleration guide.

In addition, the controller 170 may divide the required accelerationinto three or more steps to provide the user with a deceleration guidesuitable for a situation of each step.

In addition, the controller 170 may determine a condition havingpriority over the forward vehicle collision warning and control theoutput unit 130 not to output the deceleration guide even if theacceleration exceeds the determined level.

Specifically, the priority condition may be considered with priority ina case where the forward vehicle changes lanes, a case where a route isre-navigated, or in a case where an event occurs in the driving lane(e.g., lane marking) after branching.

Meanwhile, the deceleration guide may be performed in an AR screen.Specifically, the AR providing unit 160 may generate a forward vehiclecollision warning guide object and map the generated forward vehiclecollision warning guide object to a virtual 3D space to generate an ARscreen and the controller 170 may control the display unit 131 todisplay the generated AR screen.

FIG. 10 is a diagram illustrating a system network connected to a systemaccording to an exemplary embodiment of the present invention. Referringto FIG. 10, the system 100 according to an exemplary embodiment of thepresent disclosure may be implemented by various devices provided in avehicle such as a navigation device, a vehicle imaging device, asmartphone, or a device providing an AR interface for a vehicle and maybe connected to various communication networks and other electronicdevices 61 to 64.

In addition, the system 100 may calculate a current location and acurrent time zone by interworking with a GPS module according to a radiowave signal received from a satellite 20.

Each satellite 20 may transmit L-band frequencies having differentfrequency bands. The system 100 may calculate a current location basedon a time taken for the L band frequency transmitted from each satellite20 to reach the system 100.

Meanwhile, the system 100 may be wirelessly connected to a network 30through a controller station (ACR) 40, abase station (RAS) 50, an accesspoint (AP), or the like via the communication unit 180. When the system100 is connected to the network 30, the system 100 may also beindirectly connected to other electronic devices 61 and 62 connected tothe network 30 to exchange data.

Meanwhile, the system 100 may be indirectly connected to the network 30through another device 63 having a communication function. For example,when the system 100 is not provided with a module that may be connectedto the network 30, the system 100 may communicate with another device 63having a communication function through a short-range communicationmodule or the like.

FIGS. 11A and 11B are views illustrating a forward vehicle collisionwarning screen of a system according to an exemplary embodiment of thepresent invention. Referring to FIGS. 11A and 11B, the system 100 maygenerate a guide object indicating a risk of forward vehicle collisionand output the generated guide object 1001 or 1003 through AR.

Here, the guide objects 1001 and 1003 may be objects that guide a statein which user attention is required. That is, the forward vehiclecollision warning guide may be an attention guide for informing that thevehicle may collide with a forward vehicle. In addition, referring toFIG. 11B, in present exemplary embodiment, a route guide object 1002 maybe implemented as a texture image and displayed through AR. Accordingly,the driver may easily recognize the road on which the host vehicle isdriving.

In addition, the system 100 may output the guide objects 1001 and 1003through voice. Alternatively, the system 100 may output the guideobjects 1001 and 1003 through a haptic element.

FIG. 12 is a diagram illustrating an implementation form when a systemaccording to an exemplary embodiment of the present invention does nothave an image capture unit. Referring to FIG. 12, a vehicle imagingdevice 200 provided separately from the system 100 for a vehicle mayconfigure a system according to an exemplary embodiment of the presentinvention using a wired/wireless communication method.

The system 100 for a vehicle may include a display unit 131 provided onthe front of the housing, a user input unit 121, and a microphone unit123.

The vehicle imaging device 200 may include a camera 222, a microphone224, and an attachment part 281.

FIG. 13 is a diagram illustrating an implementation form when a systemaccording to an exemplary embodiment of the present invention having animage capture unit. Referring to FIG. 13, when the system 100 includesan image capture unit 150, the image capture unit 150 may capture animage of the front of the vehicle and a display part of the system 100may be recognized by the user. Accordingly, the system according to anexemplary embodiment of the present invention may be implemented.

FIG. 14 is a diagram illustrating an implementation form using a head-updisplay (HUD) according to an exemplary embodiment of the presentinvention. Referring to FIG. 14, the HUD may display an AR guide screenon the HUD through wired/wireless communication with other devices.

For example, the AR may be provided through image overlay using the HUDusing a vehicle windshield or a separate image output device, and the ARproviding unit 160 may generate an interface image overlaid on a realityimage or glass as described above. Accordingly, AR navigation or vehicleinfotainment system may be implemented.

Meanwhile, the forward vehicle collision warning guide method accordingto various exemplary embodiments of the present disclosure describedabove may be implemented as a program and provided to a server ordevices. Accordingly, each device may access the server or device wherethe program is stored, to download the program.

Meanwhile, in another exemplary embodiment, a forward vehicle detectingmethod or a forward vehicle collision warning guide method according tothe present invention may be configured by a module in a control device2100 of an autonomous vehicle 2000. That is, a memory 2122 and aprocessor 2124 of the control device 2100 may implement the forwardvehicle detecting method or the forward vehicle collision warning guidemethod in software.

This will be described in detail with reference to FIG. 15 hereinafter.

FIG. 15 is a block diagram illustrating a configuration of theautonomous vehicle 2000 according to an exemplary embodiment of thepresent invention.

Referring to FIG. 15, the autonomous vehicle 2000 according to thepresent exemplary embodiment may include a control device 2100, sensingmodules 2004 a, 2004 b, 2004 c, and 2004 d, an engine 2006, and a userinterface 2008.

In the present exemplary embodiment, the control device 2100 may includea controller 2120 including the memory 2122 and the processor 2124, asensor 2110, a wireless communication device 2130, a LiDAR 2140, and acamera module 2150.

In present exemplary embodiment, the controller 2120 may be configuredat the time when the vehicle is manufactured by a manufacturer or may beadditionally configured to perform a function of autonomous drivingafter the vehicle is manufactured. Alternatively, a component forperforming a continuous additional function may be included throughupgrading of the controller 2120 configured at the time of manufacture.

The controller 2120 may deliver a control signal to the sensor 2110, theengine 2006, the user interface 2008, the wireless communication device2130, the LiDAR 2140, and the camera module 2150 included as othercomponents in the vehicle. Although not shown, the control signal mayalso be transmitted to an acceleration device, a braking system, asteering device, or a navigation device related to driving of thevehicle.

In the present exemplary embodiment, the controller 2120 may control theengine 2006. For example, the controller 2120 may detect a speed limitof a road on which the autonomous vehicle 2000 is driving and controlthe engine 2006 to prevent a driving speed from exceeding the speedlimit or control the engine 2006 to accelerate the driving speed of theautonomous vehicle 2000 within a range not exceeding the speed limit. Inaddition, when the sensing module 2004 a, 2004 b, 2004 c, or 2004 ddetects and transmits an environment outside the vehicle to the sensor2110, the controller 2120 may receive the external environment andgenerate a signal for controlling the engine 2006 or a steering device(not shown) to control the driving of the vehicle.

When other vehicles or obstacles exist in front of the vehicle, thecontroller 2120 may control the engine 2006 or the braking system todecelerate the driving vehicle and control a trace, a driving route, anda steering angle in addition to the speed. Alternatively, the controller2120 may control the driving of the vehicle by generating a necessarycontrol signal according to recognition information of a driving lane(e.g., lane marking), a driving signal, or other external environment ofthe vehicle.

In addition to generation of the control signal by the controller 2120itself, the controller 2120 may control the driving of the vehicle byperforming communication with a nearby vehicle or a central server andtransmitting a command for controlling peripheral devices throughreceived information.

In addition, if a position or an angle of view of the camera module 2150is changed, it may be difficult to accurately recognize a vehicle orlane (e.g., lane marking) according to the present exemplary embodiment.Thus, in order to prevent this, the controller may generate a controlsignal to control the camera module 2150 to perform calibration.Therefore, in the present exemplary embodiment, since the controller2120 generates the calibration control signal to the camera module 2150,even if a mounting position of the camera module 2150 is changed due tovibration or shock generated due to movement of the autonomous vehicle2000, a normal mounting position, direction, angle of view, and the likeof the camera module 2150 may be continuously maintained. The controller2120 may generate the control signal for the camera module 2150 toperform calibration if previously stored initial mounting position,direction, angle of view information of the camera module 2150 and aninitial mounting position, direction, angle of view information, and thelike of the camera module 2150 measured while the autonomous vehicle2000 is driving are different by a threshold value or greater.

In the present exemplary embodiment, the controller 2120 may include thememory 2122 and the processor 2124. The processor 2124 may executesoftware stored in the memory 2122 according to a control signal fromthe controller 2120. Specifically, the controller 2120 may store dataand instructions for performing a lane (e.g., lane marking) markingmethod or a lane (e.g., lane marking) departure guide method accordingto the present invention in the memory 2122, and the instructions may beexecuted by the processor 2124 to implement one or more methodsdisclosed herein.

In this case, the memory 2122 may be stored in a recording mediumexecutable by the nonvolatile processor 2124. The memory 2122 may storesoftware and data through appropriate internal and external devices. Thememory 2122 may include a random access memory (RAM), a read only memory(ROM), a hard disk, and a memory 2122 device connected to a dongle.

The memory 2122 may store at least an operating system (OS), a userapplication, and executable instructions. The memory 2122 may also storeapplication data and array data structures.

The processor 2124, which is a microprocess or an appropriate electronicprocessor, may be a controller, microcontroller or state machine.

The processor 2124 may be implemented by combining computing devices,and the computing devices may include a digital signal processor, amicroprocessor or an appropriate combination thereof.

In addition, in the present exemplary embodiment, the control device2100 may monitor the inside and outside features of the autonomousvehicle 2000 and detect a state with at least one sensor 2110.

The sensor 2110 may include at least one sensing module 2004, and thesensing module 2004 may be implemented at a specific position of theautonomous vehicle 2000 according to a sensing purpose. The sensingmodule 2004 may be located at the lower, rear, front, top, or side endsof the autonomous vehicle 2000 and may also be located at an internalcomponent or a tire of the vehicle.

Through this, the sensing module 2004 may detect information related todriving such as the engine 2006, a tire, a steering angle, a speed, aweight of the vehicle, and the like as the internal information of thevehicle. In addition, the at least one sensing module 2004 may includean acceleration sensor 2110, a gyroscope, an image sensor 2110, a RADAR,an ultrasonic sensor, a LiDAR sensor, and the like and detect movementinformation of the autonomous vehicle 2000.

The sensing module 2004 may receive specific data regarding externalenvironmental conditions such as state information of a road on whichthe autonomous vehicle 2000 is located, nearby vehicle information,weather, and the like as external information, and detect acorresponding parameter of the vehicle. The sensed information may bestored in the memory 2122 depending on the purpose, either temporarilyor in the long term.

In the present exemplary embodiment, the sensor 2110 may integratedlycollect information of the sensing modules 2004 for collectinginformation generated inside and outside the autonomous vehicle 2000.

The control device 2100 may further include a wireless communicationdevice 2130.

The wireless communication device 2130 is configured to implementwireless communication between the autonomous vehicles 2000. Forexample, the autonomous vehicle 2000 may communicate with a user'smobile phone or another wireless communication device 2130, anothervehicle, a central device (traffic control device), a server, and thelike. The wireless communication device 2130 may transmit and receive awireless signal according to an access wireless protocol. A wirelesscommunication protocol may be Wi-Fi, Bluetooth, long-term evolution(LTE), code division multiple access (CDMA), wideband code divisionmultiple access (WCDMA), global systems for mobile communications (GSM)but is not limited thereto

In addition, in the present exemplary embodiment, the autonomous vehicle2000 may implement inter-vehicle communication through the wirelesscommunication device 2130. That is, the wireless communication device2130 may communicate with other vehicles on the road throughvehicle-to-vehicle communication. The autonomous vehicle 2000 maytransmit and receive information such as a driving warning and trafficinformation through vehicle-to-vehicle communication and may requestinformation or receive a request from another vehicle. For example, thewireless communication device 2130 may perform V2V communication as adedicated short-range communication (DSRC) device or a cellular-V2V(C-V2V) device. In addition to inter-vehicle communication, vehicle toeverything communication (V2X) between a vehicle and an object (e.g., anelectronic device carried by a pedestrian) may be implemented throughthe wireless communication device 2130.

In addition, the control device 2100 may include a LiDAR device 2140.The LiDAR device 2140 may detect an object around the autonomous vehicle2000 during an operation using data sensed by the LiDAR sensor. TheLiDAR device 2140 may transmit the detected information to thecontroller 2120, and the controller 2120 may operate the autonomousvehicle 2000 according to the detection information. For example, thecontroller 2120 may instruct the vehicle to reduce a speed through theengine 2006 if there is a forward vehicle driving at a low speed in thedetection information. Alternatively, the controller may instruct thevehicle to reduce an entry speed according to a curvature of a curve towhich the vehicle moves.

The control device 2100 may further include the camera module 2150. Thecontroller 2120 may extract object information from an external imagecaptured by the camera module 2150 and allow the controller 2120 toprocess corresponding information.

In addition, the control device 2100 may further include imaging devicesfor recognizing an external environment. In addition to the LiDAR 2140,a RADAR, a GPS device, a driving distance measuring device (odometry),and other computer vision devices may be used, and these devices may beselectively or simultaneously operated as needed to allow more precisesensing.

The autonomous vehicle 2000 may further include the user interface 2008for user input for the control device 2100 described above. The userinterface 2008 may allow the user to input information throughappropriate interactions. For example, the user interface 2008 may beimplemented as a touch screen, a keypad, an operation button, or thelike. The user interface 2008 may transmit an input or a command to thecontroller 2120, and the controller 2120 may perform a control operationof the vehicle in response to the input or the command.

In addition, the user interface 2008 may allow a device outside theautonomous vehicle 2000 to communicate with the autonomous vehicle 2000through the wireless communication device 2130. For example, the userinterface 2008 may interwork with a mobile phone, tablet, or othercomputer devices.

Further, in the present exemplary embodiment, the autonomous vehicle2000 has been described as including an engine 2006, but it may alsoinclude other types of propulsion systems. For example, the vehicle maybe driven by electrical energy and may be driven through hydrogen energyor a hybrid system as a combination thereof. Accordingly, the controller2120 may include a propulsion mechanism according to the propulsionsystem of the autonomous vehicle 2000 and provide corresponding controlsignals to components of each propulsion mechanism.

Hereinafter, the detailed configuration of the control device 2100 forperforming the forward vehicle detecting method or the forward vehiclecollision warning guide method according to the present invention withreference to FIG. 16 will be described in more detail.

The control device 2100 includes the processor 2124. The processor 2124may be a general-purpose single or multi-chip microprocessor, dedicatedmicroprocessor, microcontroller, programmable gate array, or the like.The processor may be referred to as a central processing unit (CPU). Inaddition, in the present exemplary embodiment, the processor 2124 may beused as a combination of a plurality of processors.

The control device 2100 also includes the memory 2122. The memory 2122may be a certain electronic component capable of storing electronicinformation. The memory 2122 may also include a combination of memories2122 in addition to a single memory.

Data and instructions 2122 a for performing the forward vehicledetecting method or the forward vehicle collision warning guide methodaccording to the present invention may be stored in the memory 2122.When the processor 2124 executes the instructions 2122 a, all or part ofthe instructions 2122 a and the data 2122 b required for the executionof the instructions may be loaded 2124 a and 2124 b onto the processor2124.

The control device 2100 may include a transmitter 2130 a, a receiver2130 b, or a transceiver 2130 c for allowing the transmission andreception of signals. One or more antennas 2132 a and 2132 b may beelectrically connected to the transmitter 2130 a, the receiver 2130 b,or each transceiver 2130 c and may include further antennas.

The control device 2100 may include a digital signal processor (DSP)2170. The DSP 2170 allows the vehicle to process digital signalsquickly.

The control device 2100 may include a communication interface 2180. Thecommunication interface 2180 may include one or more ports and/orcommunication modules for connecting other devices to the control device2100. The communication interface 2180 may allow the user and thecontrol device 2100 to interact with each other.

Various components of the control device 2100 may be connected togetherby one or more buses 2190, and the buses 2190 may include a power bus, acontrol signal bus, a status signal bus, a data bus, and the like. Thecomponents may transmit information to each other via the bus 2190 andperform a desired function under the control of the processor 2124.

The device described herein may be implemented using hardwarecomponents, software components, or a combination thereof. For example,the hardware components may include microphones, amplifiers, band-passfilters, audio to digital converters, and processing devices, Aprocessing device may be implemented using one or more general-purposeor special purpose computers, such as, for example, a processor, acontroller and an arithmetic logic unit (ALU), a digital signalProcessor, a microcomputer, a field programmable array (FPGA), aprogrammable logic unit (PLU), a microprocessor or any other devicecapable of responding to and executing instructions in a defined manner.

The processing device may run an operating system (OS) and one or moresoftware applications that run on the OS. The processing device also mayaccess, store, manipulate, process, and create data in response toexecution of the software. For the convenience of understanding, theprocessing device is described as a single processing device; however,one skilled in the art will appreciate that the processing device mayinclude multiple processing elements and multiple types of processingelements. For example, a processing device may include multipleprocessors or a processor and a controller. In addition, otherprocessing configurations such as parallel processors are also possible.

The software may include a computer program, a piece of code, aninstruction, or some combination thereof, to independent or collectivelyinstruct or combination the processing device to operate as desired,Software and data may be embodied permanently or temporarily in any typeof machine, component, physical or virtual equipment, computer storagemedium or device, or in a propagated signal wave capable of providinginstructions or data to or being interpreted by the processing device.The software also may be distributed over network coupled computersystems so that the software is stored and, executed in a distributedfashion. The software and data may be stored by one or morenon-transitory computer readable recording mediums.

The methods according to the above-described exemplary embodiments maybe recorded in non-transitory computer-readable media including programinstructions to implement various operations embodied by a computer. Themedia may also include, alone or in combination with the programinstructions, data files, data structures, and the like. The programinstructions recorded on the media may be those specially designed andconstructed for the purposes of the exemplary embodiments, or they maybe of the kind well-known and available to those having skill in thecomputer software arts. Examples of non-transitory computer-readablemedia include magnetic media such as hard disks, floppy disks, andmagnetic tape; optical media such as CD ROM disks and DVDs;magneto-optical media such as floptical discs; and hardware devices thatare specially configured to store and perform program instructions, suchas read-only memory (ROM), random access memory (RAM), flash memory, andthe like. Examples of program instructions both machine codes, such asthose produced by a compiler, and files containing higher level codesthat may be executed by the computer using an interpreter. The describedhardware devices may be configured to act as one or more softwaremodules in order to perform the operations of the above-describedexemplary embodiments, or vice versa.

The above description simply given for illustratively describing thetechnical spirit of the present invention, and those skilled in the artwill appreciate that various modifications, changes and substitutionsare possible, without departing from the essential characteristic of thepresent invention.

Accordingly, the exemplary embodiment disclosed in the present inventionand the accompanying drawings are not intended to limit but describe thetechnical spirit of the present invention, and the scope of thetechnical spirit of the present invention is not limited by theexemplary embodiment and the accompanying drawings. The scope of thepresent invention shall be interpreted by the appended claims and itshall be interpreted that all of the technical spirits in the equivalentrange are included in the scope of the present invention.

What is claimed is:
 1. A method for detecting a vehicle, the methodcomprising: receiving continuously captured front images; setting asearch area of the vehicle in a target image based on a location of thevehicle or a vehicle area detected from a previous image among the frontimages; detecting the vehicle in the search area according to a machinelearning model; and tracking the vehicle in the target image by usingfeature points of the vehicle extracted from the previous imageaccording to a vehicle detection result based on the machine learningmodel.
 2. The method of claim 1, wherein the search area is enlarged andset based on the vehicle area detected from the previous image.
 3. Themethod of claim 2, wherein the search area is enlarged and set accordingto a size of the detected vehicle.
 4. The method of claim 1, wherein thetracking comprises tracking the vehicle by extracting the feature pointsof the vehicle from the vehicle area detected from the previous image.5. The method of claim 1, wherein the location of the vehicle is trackedfrom the target image using the extracted feature points of the vehiclewhen the vehicle detection based on the machine learning model fails orreliability of the detected vehicle is below a reference.
 6. The methodof claim 1, wherein the tracking comprises tracking the vehicle inparallel with the vehicle detection in the detecting process andterminating the tracking of the vehicle when the vehicle detection basedon the machine learning model is successful or when a reliability of thedetected vehicle is above a reference.
 7. The method of claim 1, furthercomprising: displaying the detected or tracked vehicle according to apredetermined user interface.
 8. The method of claim 7, wherein thedisplaying comprises displaying a forward vehicle collision relatednotification based on the vehicle according to the predetermined userinterface.
 9. The method of claim 1, wherein the detecting furthercomprises obtaining a motion vector of the feature points of the vehiclefrom a plurality of previous images and generating a modified searcharea based on the motion vectors and the search area and comprisesdetecting the vehicle from the modified search area according to themachine learning model.
 10. The method of claim 9, wherein the motionvector is generated based on a relationship between positions at whichthe feature points of the vehicle are expressed in each of the pluralityof previous images.
 11. The method of claim 9, wherein a center positionof the modified search area is determined based on a center position ofthe search area and the motion vector, and a width of the modifiedsearch area is determined based on a direction or a size of the motionvector.
 12. A vehicle detecting apparatus comprising: an image inputunit receiving continuously captured front images; an area setting unitsetting a search area of a vehicle in a target image based on a locationof the vehicle or vehicle area detected from a previous image among thefront images; a vehicle detecting unit detecting the vehicle from thesearch area according to a machine learning model; and a vehicletracking unit tracking the vehicle in the target image using featurepoints of the vehicle extracted from the previous image according to avehicle detection result based on the machine learning model.
 13. Thevehicle detecting apparatus of claim 12, wherein the search area isenlarged and set based on the vehicle area detected from the previousimage.
 14. The vehicle detecting apparatus of claim 13, wherein thesearch area is enlarged and set according to a size of the detectedvehicle.
 15. The vehicle detecting apparatus of claim 12, wherein thevehicle tracking unit tracks the vehicle by extracting feature points ofthe vehicle from the vehicle area detected from the previous image. 16.The vehicle detecting apparatus of claim 12, wherein the vehicletracking unit tracks the location of the vehicle from the target imageusing the extracted feature points of the vehicle when the vehicledetection based on the machine learning model fails or a reliability ofthe detected vehicle is below a reference.
 17. The vehicle detectingapparatus of claim 12, wherein the vehicle tracking unit tracks thevehicle in parallel with the vehicle detection in the detecting processand terminates the tracking of the vehicle when the vehicle detectionbased on the machine learning model is successful or when a reliabilityof the detected vehicle is above a reference.
 18. The vehicle detectingapparatus of claim 12, further comprising: an output unit displaying thedetected or tracked vehicle according to a predetermined user interface.19. The vehicle detecting apparatus of claim 17, wherein the output unitdisplays a forward vehicle collision related notification based on thevehicle according to the predetermined user interface.
 20. A method ofwarning a vehicle rear-end collision, the method comprising: receivingcontinuously captured front images; setting a search area of the vehiclein a target image based on a location of the vehicle or a vehicle areadetected from a previous image among the front images; detecting thevehicle in the search area according to a machine learning model;tracking the vehicle in the target image by using feature points of thevehicle extracted from the previous image according to a vehicledetection result based on the machine learning model; and determining apossibility of rear-end collision according to a distance to thedetected or tracked vehicle and a relative speed.